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Welcome and introductions 


• Who am I? 

Janice Tripney, Lecturer in Social Policy, 
EPPI-Centre, Institute of Education, 
University College London (UCL), UK. 

• Who are you? 

• What we will be discussing today? 
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On completion participants will.. 


• have a basic understanding of the purposes of systematic 
reviews and their relevance to evidence-informed policy and 
practice 

• have a general overview of the standard stages of the review 
process common to many reviews 

• have discussed how reviews vary and the relationship 
between evidence maps, rapid evidence assessments and 
systematic reviews 

• have addressed key issues to consider when commissioning 
or using a review 

• be aware of infrastructure developed to support the 
production and use of reviews 
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1. The what and why of being 
systematic 
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Systematic reviews are pieces of 
research in their own right 
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What makes a review of evidence 
systematic? 


• It uses a formal 

accountable method for 
bringing together what 
we ‘know’. 

• Has explicit methods 
regarding design and 
execution. 

• Is transparent (e.g. in its 
criteria, decisions, 
reporting). 
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Why is systematic reviewing 
necessary? 


To reduce the influence of any flaws or errors in a single 
study. 

There may be so much literature on a topic that it is 
impossible to make sense of it. 

Research often produces results that appear to be 
contradictory, even when they are actually consistent. 

Research often produces results that are actually 
contradictory (but at first sight they may not seem so). 
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How do systematic reviews differ 
from literature reviews? 


• They use explicit and transparent methods and follow a 
standard set of stages. This enables them to be 
replicated and updated - and so provide flexibility and 
value for money in the longer term. 

• They answer questions asked by people who want to use 
the answers. 


• The process allows for different studies to be weighted 
for quality and relevance of evidence for a given 
question. 

• They are undertaken by teams and the outcome is a 
collective one - which reduces potential bias. 



2. Common stages of the systematic 
review process 


I. Form review team (involve ‘users’) 

II. Set the scope of the review (formulate review question, 
conceptual framework and inclusion criteria = develop 
‘protocol’) 

III. Search for and identify relevant studies 

IV. Describe studies 

V. Afesess study quality (and relevance) 

VI. Sunthesise findings 

VII. Communicate and engage 
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i. Form review team 


Systematic reviews are always a team effort. 
Who might you have in your team? 

- Content experts 

- Systematic review methods experts 

- Statistician 

- Librarian 

- Review users 
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What can each member bring? 


Content experts: provide expertise in the area covered by the 
review (e.g. practitioners and academics). 

Systematic review methods experts: helps to have someone 
familiar with the procedures and documentation standards for a 
systematic review. 

Statistician: If meta-analysis is to be considered, access to a 
statistician with relevant experience is needed. 

Librarian: experience with the extensive searching and 
documentation procedures; responsible for maintaining the 
database of references. 

Review users: ? 
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Why involve review users in 
systematic reviews? 


• To broaden perspectives (to democratise 
knowledge / to give voice to marginalised 
groups). 

• To reduce/uncover researcher 
assumptions and biases via a transparent 
and critical approach: 

- What questions are being asked? Why? 

- Who and what is influencing the way an 
issue is looked at? 

• To improve quality and relevance. 

• To improve accessibility by translating 
jargon. 

• To encourage wider dissemination of the 

review. 
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Bottom line: to increase the likelihood of 
producing reviews that are relevant and useful. 















ii. Set the scope of the review 


• Review question 

• Conceptual framework 

• Selection criteria 
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Setting the scope: the review 
question 


• Is the driver for all review processes 

• Is an investigative statement rather 
than a topic of interest 

• Should be clear and answerable 

• Takes time to develop (and there are 
tools/frameworks to help with this) 
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Common constructs in review 
questions 


• Population, problem or phenomenon 

• Intervention or exposure 

• Outcome(s) 

• Comparison or control condition 

• Context or environment or setting 

• Professionals, practitioners or providers 

• Research (study design or empirical methods) 

• Stakeholders or perspectives 

• Timeframe or duration 
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Example of ‘effectiveness’ question 
(using just 3 constructs) 


What effect has electrification had on health, 
welfare and education outcomes in LMICs. 
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Tools to assist with question 
formulation 


ECLIPSE (Wildridqe & Bell 2002) 

Expectation - Client group - Location - Impact - Professionals involved - Service 

This framework is useful for questions relating to health policy and management issues. Expectation encourages 
reflection on what the information is needed for i.e. improvement, innovation or information. Impact looks at what 
you would like to achieve e.g. improve team communication. 


EPICOT (Brown et al 2006) 

Evidence - Population - Intervention - Comparison - Outcome - Timestamp 

Designed to address research recommendations on the effect of treatments. In addition to the PICO elements, 
Evidence is for the current state of the evidence and Timestamp is for the date of the recommendations. 


PECODR (Dawes et al 2007) 

Population - Exposure - Comparison - Outcome - Duration - Results 

Useful for case control studies and cohort studies. Duration can be used to clarify the length of the follow up period 
and the Results could be used for Number Needed to Treat or similar. 


SPICE (Booth 2006) 

Setting (context) - Perspective- Intervention - Comparison - Evaluation 

Useful for qualitative studies that seek to evaluate a service. Perspective relates to users or potential users. 
Evaluation is how you plan to measure the success of the intervention. 


SPIDER (Cooke. Smith & Booth 2012) 

Sample - Phenomenon of Interest - Design (of study) - Evaluation - Research type 

Useful for qualitative or mixed methods research. Phenomenon of Interest includes behaviours and/or experiences 
e.g. compliance. 
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Setting the scope: the conceptual 
framework 


• Definitions of all key concepts (e.g. 
population, intervention, outcomes) to be 
studied, and the relationships between 
them. 

• Requires considerable thought and 
discussion to explore ideas and challenge 
theoretical assumptions. 

• Influences all stages of the review. 
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Setting the scope: selection criteria 


• Specify the nature of the studies that will be 
included in the review (e.g. topic, method, date, 
type of publication, language). 

• If you have a well-defined question and 
conceptual framework the selection 
(inclusion/exclusion) criteria will be obvious 
and easy to formulate. 
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Example of selection criteria 


Draft inclusion criteria for a systematic map with the question: ‘what are the 
barriers to and what are the facilitators of healthy eating in primary 
schools?’ 

Studies should be included if they meet aN of the following criteria (i-viii): 

(i) study is focused on healthy eating in schools; and 

(ii) study is focused on children or young people aged four to eleven years, or aspects of 
their lives; and 

(iii) study is about the promotion of healthy eating, or the barriers to, and facilitators of, 
healthy eating; and 

(iv) study includes participants from an OECD* country; and 

(v) study has a design** that either 

(a) evaluates the outcomes or processes of an initiative or programme (an intervention) aimed at 
promoting healthy eating in schools; or 

(b) is a systematic review of such evaluation studies; or 

(c) explores how various aspects of children’s or young people’s lives are associated with eating 
healthily in schools; or 

(d) is a systematic review of studies of variables associated with eating healthily in schools; or 

(e) studies the views of people about healthy eating in schools; or 

(f) is a systematic review of studies of people’s views; and 

(vi) study is published in English; and 

(vii) study is published in or after 1998; and 
(viii) study is not an unpublished thesis. 
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iii. Search for relevant studies 
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Systematic searching: common 
characteristics 


Search methods are: 

- intended to find studies that might be 
relevant for the review’s question. 

- derived from the review question 

- practically constrained 

- supported by a rationale 

- explicitly reported 

- iterative: draft, test and implement a 
structured search plan (search strategy). 
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Types of sources of research 


Search strategy is also detailed in full in the Review 
Protocol. 

Don’t rely on databases alone. 

• Bibliographic databases (and library catalogues) 

- ‘General’ (e.g. ERIC, Medline, EconLit) 

- ‘Specialised’ (e.g. OpenGrey) 

• Internet search engines and gateways 
(e.g. Google Scholar) 

• Hand-searching websites (of NGOs etc.) 

• Hand-searching journals 

• Scanning reference lists (backward citation checking) 

• Forwards citation searching 

• Professional contacts, key authors/ experts, networks 
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The basics of bibliographic 
databases 


• Use a range of different terms for each 
concept in your review and combine with 
Boolean operators (AND/OR) 
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Concepts and search terms 



Andrews et al. 
(2002) A 
systematic 
review of the 
impact of 
networked ICT 
on 5-16 year 
olds’ literacy in 
English. 
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Concepts and search terms 


An example search strategy for one database. Taken from Andrews et al (2002) A systematic review of the impact of 
networked ICT on 5-16 year olds’ literacy in English. Research Evidence in Education Library. London: EPPI-Centre 


1. explode "Computers" 

2. explode "Computer-Applications" 

3. "Computer-Games" in DE 

4. explode "Computer-Simulation" 

5. explode "Computer-Software" 

6. "Electronic-Communication" in DE 

7. explode "Information-Systems" 

8. "Internet-" in DE 

9. "Word-Processing" in DE 

10. #1 or #2 or #3 or #4 or #5 or #6 or #7 or #8 or #9 

11. "Literacy-" in DE 

12. "Literacy-Programs" in DE 

13. explode "Language-Arts-Education" 

14. explode "Reading" 

15. "Reading-Development" in DE 

16. explode "Reading-Measures" 

17. explode "Reading-Skills" 

18. "Writing-Skills" in DE 

19. #11 or #12 or #13 or #14 or #15 or #16 or #17 or #18 

20. "Computer-Assisted-Instruction" in DE 

21. #10 or #20 

22. #21 and #19 


23. (ict near (literacy or read or reading or spell or spelling 
or write or writing)) in ti,ab 

24. (information technolog* near (literacy or read or reading or 
spell or spelling or write or writing)) in ti,ab 

25. (communication technolog* near (literacy or read or reading 
or spell or spelling or write or writing)) in ti,ab 

26. ((cal or cai or networked technolog*) near (literacy or read 
or reading or spell or spelling or write or writing)) in ti,ab 

27. ((multimodal or digital media or internet) near (literacy or 
read or reading or spell or spelling or write or writing))in ti,ab 

28. ((cdrom or hypertext or wide web or www or worldwide web) near 

29. ( (software or computer* or electronic) near (literacy or 

read or reading or spell or spelling or write or writing))in ti,ab 

30. #23 or #24 or #25 or #26 or #27 or #28 or #29 

31. "Adult-Development" in DE 

32. "Adult-Education" in DE 

33. "Adult-Learning" in DE 

34. exact{ADULTHOOD} in AG 

35. exact{AGED} in AG 

36. exact{MIDDLE-AGE} in AG 

37. exact{THIRTIES} in AG 

38. exact{VERY-OLD} in AG 

39. exact{YOUNG-ADULTHOOD} in AG 

40. #31 or #32 or #33 or #34 or #35 or #36 or #37 or #38 or #39 

41. #22 or #30 

42. #41 not #40 

43. #42 and (LA = "ENGLISH") 


(literacy or read or reading or spell or spelling or write or writing)) 


in ti,ab 
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Specialist databases and library 
catalogues (reported in protocol 
appendix) 
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Websites (reported in protocol 
appendix) 


AF 7 ENDIX 3 : WEBSITES 
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iii. Identify relevant studies 
(screening) 


Process of deciding 
which studies meet the 
inclusion criteria. 

Titles and abstracts 
screened first, full paper 
screened only if study 
still looks relevant. 

Important to be over- 
inclusive initially, then 
apply criteria more 
strictly to full papers. 
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How is screening commonly done? 


:E3 «» GO A 


Auto advance 


O (jj Screening on Title and Abstract 
□ EXCLUDE on date | info | 

@ EXCLUDE on target group | info 
I | EXCLUDE on intervention | info 
0 EXCLUDE on evidence info | 

Q EXCLUDE on country | info | 

I I INCLUDE for second opinion | info 
Q INCLUDE on abstract | info J 
O (ji Screening on Full Report 
A O |j) Demo keywording tool 
jh O / Demo data extraction tool 
^ / Allocation codes 
^ O / Retrieval status 
jh O / Report sets 


Citation details Reference Search Coding record Linked records 


Item 1 / 262 


Previous Next 


4 Save and close Save Cancel 


Title Aflatoxin B< and ethanol co-exposure induces hepatic oxidative damage in mice. 


Author(s) | Adedara I A A; Qwumi SEE; Uwaifo A O O; Farombi E (| Item IDs Internal: 1224414 Imported: 

Month | September j Pub type | Journal, Article 


Year 2010 


Included? 0 


The present study investigated the effects of aflatoxin Bi (AFBi) and ethanol co-exposure on biomarkers of hepatic damage in 
mice. Four groups of adult male mice were treated for 7 consecutive days. Control mice received com oil alone at a dose of 2 
mL/kg bw. One group was treated with ethanol at a dose of 500 pL/kg bw and another group administered 9 mg/kg bw of 
AFBi dissolved in com oil. The fourth group was co-administered with ethanol and AFBi. The body and liver weights of treated 
mice decreased significantly when compared with corresponding control. Alone, ethanol and AFB(l) treatment separately 
increased serum activities of aspartate aminotransferase (AST), alanine aminotransferase (ALT), gamma glutamyl transferase 
(GGT) and alkaline phosphatase (ALP). Alcohol dehydrogenase (ALD) activity was markedly elevated in ethanol-treated mice 
but was unaffected by AFBi treatment. Co-exposure of AFBi and ethanol escalated the activities of these serum enzymes. 
Administration of ethanol and AFBi separately resulted in significant decrease in both non-enzymatic antioxidant glutathione 
(GSH) level and enzymatic antioxidant catalase (CAT) and glutathione-S-transferase (GST) activities, whereas lipid 
peroxidation was markedly elevated. Superoxide dismutase activity and vitamin C level remained unaffected in all treatment 
groups. Co-exposure of animals to ethanol and AFBi showed additive effects on the activities of GST and CAT as well as on the 
GSH level. Histopathological study revealed that these compounds interact together to exacerbate their individual effects on 
the liver. In summary, the data presented showed that AFBi and ethanol co-exposure induced severe oxidative damage to the 
liver of mice and as such humans consuming excessive amount of ethanol and diets contaminated with AFBi simultaneously 
may be at greater risk of the hepatotoxic effects of these compounds. 


Journal title Toxicology and industrial health 


Pages [717-24 


Short title | Adedara (2010) 
Edition 


URL | http://www.ncbi.nlm.nih.gov/pubmed/20837563 _| ISBN/ISSN | ISSN: 1477-0393 
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iv. Describe studies 


• Different products require different data 

• Basic data to describe what research has 
been done and how i=^> map 

• Detailed data on methods and results 

i=> synthesis 
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Data extraction guidelines 


• Each study is interrogated using the same set of 
questions 

• ‘Answers’ in data extraction (coding) guidelines consist of 

pre-set categories and/or open text boxes 

• Often contains guidance (a manual) to help reviewers 
interpret questions or answers 

• Different reviewers and review teams often have their own 
data extraction guidelines 

• Variation between generic, discipline-specific and review- 
specific guidelines. 


( 32 ) 



Example of data extraction 
guidelines 


Coding guidelines are also detailed in full in the Protocol. 
This one is over 10 pages 
Here is the first two sections. 


APPENDIX 42 CODING TOOL 

QuBtfau 

Axmve r* 

SectiDfi A: Core keywi rd* 


A l Name eJ redtfwer 

A.i.J Details ij specify} 


A.i.3 Nans kn::wn 

A 2 Linked repoirLx 

A.2.2 Linied. (specify} 


A.i.J UucImij [spedFyi 

Sertaan D: Si udy characteristics 


B.i_a Jaixmal adide 

B.i Farm af piftiLj^ban 

B.i 2 Tadimcal repo id (specify! 

Douroertal iixii /thesis [jpedfyJ 


Bi 4 Other f specify! 


B.2.S ii» 4 - lWJ 

B.2, Y“ir ixf puhlidliar. 

B. 2.2 2000-2004 

B.2.3, aoqj-am^ 


B.2-4 2010-2014 


B.3.S Nat stated 

B.3 Study funding 

B.3.2 Slated (KpadBiO 


B. J.3, Undear (specify 5 


B.4.3 ]jm--in:xime ciiunliy (spedfyi 

B-4 5-1 udy setling 

B.4.2 Li/wer-middle Lncame camntry [specif) 


B.4..3, Upper-middle incurne cmintry (sqieciftO 


B.44. I iigli -nccmce ■nnmtry (specify] 


E.J-u r^il Asin and Pacific (specify} 


B.J 41 Eurupe and Central . 4 .-: La (jpe-riFy! 

B.£ Tlegj.ani 

B.^g Latin America and. Carilhbsin (spsdfy} 

B.!:..5 Middle TLail and Nart-h Africa (spedfy) 


B.Jkiulb Asia (specify] 


B.ijjfi ff nib-, Soli aran. Africu (specify! 

SectiiDfi C : Study simple cHaor act eristic* 


C.i L'idt 4 if •: 1 his e rval i •: i n 

C.l_i Staled [specify! 

C.L2 Undear (aped .fy] 
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v. Assess study quality (and 
relevance) 


Why appraise research studies as part of a systematic 
review? 


Studies may vary quite significantly in: 

- quality of individual studies 

- relevance of studies to a particular review question 


Reviewers want to ensure that the most trustworthy and 
relevant studies are used to develop the conclusions of the 
review. 
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A number of reviews have found that lower quality studies 
tend to overestimate the effects of interventions. 



What is critical appraisal? 


An approach used (in systematic reviews 
and elsewhere) to identify variation in 
quality and relevance between studies. 

Also referred to as ‘quality assessment’ or 
‘risk of bias assessment’. 
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Critical appraisal tools 


Given the number of tools available, reviewers are likely to be 
faced with the question of ‘which tool should I use?’ 

• Ideally you want tools that are: 

- specific to the study designs being evaluated 

- address methodological rigour (according to accepted 
standards for particular study design) 

- based on empirical evidence of bias 
CASP http://www.casp-uk.net/ 

Cardiff University resources http://www.cardiff.ac.uk/specialist-unit- 
for-review-evidence/resources/critical-appraisal-checklists 

Cochrane Risk of Bias Tool 

http://methods.cochrane.org/bias/assessina-risk-bias-included- 

studies#The%20Cochrane%20Risk%20of%20Bias%20Tool 
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Use of critical appraisal 


Reviewers want each 
study to contribute 
appropriately to the 
review’s findings 

But how can study ratings 
be used? 

- Exclude low quality studies? 

- Weight them differently? 

- Examine differences in findings 
according to quality (sensitivity 
analysis) 

- Leave it to the reader? 



“• ftp*" ihtrvi a 40% chance of thowert, and a 10% 
chance we know what we're talking about." 
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vi. Synthesise findings 


‘The process or result of building up separate 
elements, especially ideas, into a connected whole, 
especially a theory or system’ (Oxford English 
Dictionary) 

• So, it’s a a process of building separate elements 
into something new. 

• Not just a report of the findings of the individual 
studies in a review. 

• Involves a transformation of the data from 
primary studies in some way. 
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Many configurative ‘approaches’ to 
synthesis 


• Meta-ethnography 

• Thematic synthesis 

• Textual narrative synthesis 

• Framework synthesis 

• Grounded theory 

• Critical interpretive synthesis 

• Meta-study 

• Meta-narrative 

• Ecological Triangulation 
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See: Barnett-Page E, Thomas J (2009) Methods for the synthesis of qualitative research: a critical 
review . BMC Medical Research Methodology 9: 59. doi:10.1186/1471-2288-9-59 





vii. Communicate and engage 


Important issues that researchers 
conducting systematic reviews should 
consider: 

• Report writing 

- Format (1-4 page policy brief, short report, technical 
report?) 

- Language (user-friendly, not overly technical) 

• Dissemination plans 

- Draw on ‘users’ in Advisory Group 
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Group exercise 


• Think of a review question that you are 
interested in having answered. 
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Group exercise 


• Think of a review question that you are 
interested in having answered. 

• What challenges do you think stand in the 
way of a systematic review being 
undertaken to answer this question? 
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3. Dimensions of difference in 
reviews (i.e. how reviews vary) 


The task of developing a system for classifying or 
categorising reviews types is, at the moment, rather 
challenging. 

There is no consensus on terminology and labels used. 

Some of the key dimensions on which reviews differ: 

- Questions and conceptual framework. 

- Studies considered. 

- Single or multi-component reviews. 

- Methods of review, and aggregative and/ or configuring 
emphasis (Sandelowski et al., 2012). 

- Breadth, depth and time available. 
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4. Evidence maps and rapid evidence 
assessments 
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What is a systematic map (AKA 
evidence map) and why map the 
research field? 


• Mapping of research activity describes what 
research has been undertaken in a field. 


• Purposes: 

- a useful product in its own right; 

* Useful to policymakers and practitioners looking for 
evidence to inform policies ana programmes. 

• For donors and researchers, maps can help identify areas 
where there is an urgent need for more rigorous research 
evidence. 

- to plan for synthesis (to allow narrowing of the research 
question and the criteria for studies included in the 
synthesis) 
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Examples of systematic maps 
(EPPI-Centre) 

http;//eppi.ioe.ac.uk/cms/Default.aspx?tabid=2462 


Links 

Reviews facility home 
Technical report (pdf] 


A systematic map of the research on the relationship between obesity 
and sedentary behaviour in young people 

What do we want to know? 

The systematic map aimed to review the scope of the quantitative research literature on the relationship between 
obesity and sedentary behaviour in young people aged 6-16 in order to: 


Download report 
(pdf) 



■ Identify where there appeared to be gaps in the literature; 

■ Assess whether there was sufficient research to warrant an in-depth review and synthesis. 

Who wants to know and why? 

In January 2008, in the report 'Healthy Weight, Healthy Lives', the Government recognized obesity reduction as a 
national priority and set out an aim to reverse the current obesity trend, It has been argued that a better 
understanding of the dynamics of weight gain amongst young people is required to design effective policy solutions. 
Increasing levels of sedentary behaviour such as television viewing have been hypothesized as having some relation 
to the increase in weight being found amongst young people, 



What did we find? 

■ 326 relevant studies were identified, The majority of studies were conducted in the United States, followed by 
Canada and Australia. Only 15 were undertaken in the UK. Studies covered both Primary and Secondary school 
age ranges and most covered both boys and girls. 

■ Studies used a variety of different measures of sedentary behaviour, The most frequent measure was time spent 
in TV, film and video viewing or a proxy for this, such as TV in the bedroom or number of TVs in the house. This 
was followed by computer use/playing computer games/owning a computer and playing video games. Most 
studies use Body Mass Index (BMI) as an indicator of obesity. 

■ Most studies explored the relationship between obesity and sedentary behaviour. 197 studies attempted to assess 
the effect of sedentary behaviour on obesity, 65 studies aimed to assess the effect of obesity on sedentary 

hphauinnr anH piniht dnirfip^ aQQPQQpH bnfh Tn aHHil-inn thprp wptp 41 infprwpntinn tfiiriipG fhaf affprnnfpH fn 
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Examples of systematic maps (3ie) 

http://www.3ieimpact.org/en/evidence/gap-maps/ 


• 3ie evidence gap maps (EGMs) consolidate what we know 
about what works in particular development sectors or thematic 
areas. 

• 3ie’s EGMs identify evidence from systematic reviews and 
impact evaluations and provide a graphical display of areas 
with strong, weak or non-existent evidence on the effects of 
development programmes and initiatives. 

• 3ie has recently developed a new interactive and dynamic 
online platform w hich allows users to explore the evidence 
included in a particular EGM, with links to user-friendly 
summaries of all studies. 

• To know more about evidence gap maps and how to 
conduct them, click here . 

• To get some tips on how to navigate our new dynamic 
evidence gap map platform, download the how to use 
evidence gap map document (229.7 KB) 
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3ie: Water, Sanitation and Hygiene 
Evidence Gap Map 
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Click items in the legend to toggle the category off and on in the graph. High, Medium and Low Confidence and Protocol categories apply only to Systematic Reviews. 
High, Medium and Low Confidence refersto confidence in conclusions about effects. It indicates the overall rating given to a systematic review based on a careful 
appraisal of the methods applied in a systematic review, using a standardised checklist. 


























Rapid evidence assessments (AKA 
rapid reviews) 


• When time and resources are limited, reviewers can 
reduce: 

scope of the review 
sources searched 
sensitivity of search 

- data coded 

- quality assurance measures 

- depth of analysis 
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Trade-off between speed of execution and: 


- depth of analysis 
breadth of coverage 
reliability 

When does a rapid review become a 

non-systematic scoping review? 



This PROJECT is doe TOMORROW, With 
NO BUDGET, NO SUPPORT STAFF AND NO 
GUIDELINES. NOW IS YOUR chance TO 
IMPRESS ME! 


9iada yiw 











5. Key issues to consider when 
commissioning or using a review 
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Using reviews: issues to consider 


1. Where to find existing systematic reviews of 

evidence? (specialist databases - e.g. Campbell Collaboration, 
3ie, EPPI-Centre, CRD; and via commercial databases using 
methodological filter) 

2. Quality of the systematic review 

3. How will they be used? 

• Identify areas to study in new research 

• Help design new studies 

• Grant applications 

• Inform 

- Practitioners 

- Policy-makers 

- Consumers/clients 

4. Other factors (research evidence not the only factor in decision 
making) 



Commissioning reviews: issues to 
consider 


1. Is there a need for a new systematic review? 

- Do some investigative work! 

2. How will you ensure the review question is going to 
help address the policy/practice need? 

3. If putting out a ‘call for tenders’, is there in-house 

expertise to assess the quality of the bids? 

4. If an in-house team of researchers are to conduct the 
review, do they have the right mix of skills? 

- For 2, 3 and 4, consider linking up with one of the specialist organisations offering 
support. 
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6. Infrastructure developed to support 
the production and use of reviews 


• Campbell Collaboration www.campbellcollaboration.org 

• Cochrane Collaboration www.cochrane.org 

• Collaboration for Environmental Evidence (CEE) http://www.environmentalevidence.org/ 

Non-profit international research networks that produce, support and disseminate systematic reviews 

• Joanna Briggs Institute http://ioannabriggs.org/ 

A not-for-profit research and development organisation at the University of Adelaide 

• EPPI-Centre http://eppi.ioe.ac.uk/cms/ 

A university-based research organisation based at University College London 

• Global Evidence Synthesis Initiative (GESE) http://epoc.cochrane.org/news/gesi 

An international consortium established to enhance the capacity for research synthesis worldwide 

• African Evidence Network (AEN) http://www.africaevidencenetwork.org/resources/ 

A community of people who work in Africa and have an interest in evidence, its production (in particular but not 
exclusively through systematic reviews) and use in decision-making. 

• International Initiative for Impact Evaluation (3ie) http://www.3ieimpact.org/en/funding/svstematic-reviews- 
orants/ 

3ie has partnered with the Campbell Collaboration to provide training and support to review authors. 
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Focused on c apacity-building , the above organisations offer a range of training and support services, including 
online resources. 














7. Further resources 


( 55 ) 



Further reading (setting the scope) 


• Armstrong A, Hall B, Doyle J, Waters E (2011 ) ‘Scoping the scope’ of a cochrane 
review. Journal of Public Health 33(1): 147-150. 

• Cooke A, Smith D, Booth A (2012) Beyond PICO: the SPIDER tool for qualitative 
evidence synthesis. Qual Health Res 22(10):1435-43. 

• Davies K (2011) Formulating the Evidence Based Practice Question: A Review of 
the Frameworks. Evidence Based Library and Information Practice 6(2) : 75-80. 

• Guidelines for preparation of review protocols. 

http://www.campbellcollaboration.Org/artman2/uploads/1/C2 Protocols guidelines vl.odf 

• Oliver S, Dickson K and Newman M (2012) Getting started with a review. In: Gough D, 
Oliver S, Thomas J (Eds) An Introduction to Systematic Reviews. London: Sage. 

• Search Concept Tools 

http://medhealth.leeds.ac.uk/info/639/information specialists/1500/search concept tools 

• Weir M, Grimshaw J, Mayhew A, Fergusson D (2012) Decisions about lumping vs. 
splitting of the scope of systematic reviews of complex interventions are not well 
justified: a case study in systematic reviews of health care professional reminders. 

J Clin Epidemiol 65(7):756-63. 


( 56 ) 






Further reading (searching) 


• Brunton G, Stansfield C, Thomas J (2012) Finding relevant studies. In: An introduction to systematic reviews. (Eds) 
Gough D, Oliver S, Thomas J. London: Sage. 

• Hausner E, Guddat C, Hrmanns T et al (2014) Development of search strategies for systematic reviews: 
validation showed the non-inferiority of the objective approach. J Clinical Epidemiol [Epub ahead of print] 

• Methley AM, Campbell S, Chew-Graham C et al (2014) PICO, PICOS and SPIDER: a comparison study of 
specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res 4(1): 
579. [Epub ahead of print] 

• McGowan J, Sampson M, Lefebvre C (2010) An Evidence Based Checklist for the Peer Review of Electronic 
Search Strategies (PRESS EBC) Evidence Based Library and Information Practice 2010, 5.1 

http://eiournals.librarv.ualberta.ca/index.php/EBLIP/article/view/7402/6436 

• Schucan Bird K and Tripney J (2011) Systematic literature searching in policy-relevant, interdisciplinary 
reviews: an example from culture and sport. Research Synthesis Methods 2(3): 163-173. 

• Stansfield C, Brunton G, Rees R (2014) Search wide, dig deep: literature searching for qualitative research. An 
analysis of the publication formats and information sources used for four systematic reviews in public health. 

Research Synthesis Methods 5:142-151. 

• Taylor B, Dempster M and Donnelly M (2003) Hidden gems: systematically searching electronic databases for 
research publications for social work and social care. British Journal of Social Work 33:423-429. 

• Thomas J, McNaught J, Ananiadou S (2011) Applications of text mining within systematic reviews. Research 
Synthesis Methods 2:1-14. 

• Shemilt I, Simon A, Hollands G et al (2013) Pinpointing needles in giant haystacks: use of text mining to reduce 
impractical screening workload in extremely large scoping reviews. Research Synthesis Methods 5: 31-49. 
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Further reading (data extraction) 


• Eppi-Centre keywording/data extraction guidelines 
http://eppi.ioe.ac.Uk/cms/Default.aspx7tabids184#Guidelines 

• Elamin MB, Flynn DN, Bassler D et al (2009) Choice of data extraction tools 
for systematic reviews depends on resources and review complexity. J 

Clin Epidemiol 62(5):506-10. 

• Higgins JPT, Deeks JJ, (eds). Ch 7: Selecting studies and collecting data. 

In: Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of 
interventions. Version 5.0.0: The Cochrane Collaboration. Available from: 

www.cochrane-handbook.org 

• Oliver S and Sutcliffe k (2012) Describing and analysing studies. In: An 
introduction to systematic reviews. (Eds) Gough D, Oliver S, Thomas J. 
London: Sage. 

• Petticrew M and Roberts H (2006) Appendix 4: example data extraction 

form. In: Systematic reviews in the social sciences: a practical guide. Oxford: 
Blackwell publishing. 
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Further reading (appraising studies) 


• Critical Appraisal Skills Programme http://www.casp-uk.net/ 

• Harden A & Gough D (2012) Quality and relevance appraisal. In: An introduction to 
systematic reviews. (Eds) Gough D, Oliver S, Thomas J. London: Sage. 

• Spencer L, Ritchie J, Lewis J et al (2003) Framework for appraising the quality of 
qualitative evaluations. London: Cabinet Office 

http://www.civilservice.aov.Uk/wp-content/uploads/2011/09/a quality framework tcm6- 

38740.pdf 

• Petticrew M & Roberts H (2009) How to appraise the studies: an introduction to 
assessing study quality. In: Systematic reviews in the social sciences: a practical 
guide. Oxford: Blackwell Publishing. 

• Schunemann H, Best D, Vist G et al (2003) Letters, numbers, symbols and words: 
how to communicate grades of evidence and recommendations. CMAJ 169 (7): 
677-680 

• West S, King V, Carey T et al (2002) Systems to Rate the Strength of Scientific 
Evidence. Evidence Report/Technology Assessment No. 47 AHRQ Publication No. 
02-E016. Rockville, MD: Agency for Healthcare Research and Quality. 
http://c.vmcdn.com/sites/www.enerqypsvch.orq/resource/resmqr/imported/Svstems%20 

To%20Rate%20The%20Strength%20of%20Scientific%20Evidence%20-%20AHRQ.pdf 
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Further reading (how reviews vary) 


• Dixon-Woods M, Agarwal S, Jones D et al (2005) Synthesising qualitative and quantitative 
evidence: a review of possible methods. J Health Serv Res Policy 10:45-53.. 

• Gough D and Thomas J (2012) Commonality and diversity in reviews. In: Gough D, Oliver S, 
Thomas J (Eds) An Introduction to Systematic Reviews. London: Sage. 

• Gough D, Thomas J, Oliver S (2012) Clarifying differences between review designs and 

methods. Systematic Reviews 1:28. 

• Gough D (2013) Meta-narrative and realist reviews: guidance, rules, publication standards and 
quality appraisal. BMC Medicine 11: 22. 

• Grant M and Booth A (2009) A typology of reviews: an analysis of 14 review types and 
associated methodologies. Health Information & Libraries Journal 26(2):91-108. 

• Mallidou A (2014) Mapping the landscape of knowledge synthesis. Nursing Management 21 (5): 
30-39. 

• Sandelowski M, Voils C, Leeman J et al (2012) Mapping the Mixed Methods-Mixed Research 
Synthesis Terrain. J Mix Methods Res 6(4): 317-331. 

• Thomas, James; Newman, Mark; Oliver, Sandy (2013) Rapid evidence assessments of research 
to inform social policy: taking stock and moving forward. Evidence & Policy 9(1): 5-27. 

• Woodman J, Thomas J, Dickson K (2012) How explicable are differences between reviews that 
appear to answer a similar question? Systematic Reviews 1: 37. 
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EPPI-Reviewer (reviewing software) 


You can start using EPPI-Reviewer 4 today by 
signing up for a free one-month trial: 

http://eppi.ioe.ac.uk/cms/Default.aspx?alias=eppi.ioe 

.ac.uk/cms/er4 


A user manual and instructional videos can be found 
here: 

http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=2933 
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